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ihc  infomiiil  observation  that  photographs  with  smoothly  changing  albedo  arc  usually  intcrpr.ncd  as 
having  smoothly  changing  topography. ) 

llic  concern  here  is  whether  there  can  be  zero-crossings  corresponding  to  topographic  rather 
tJian  photometric  changes  in  an  object,  for  example,  in  the  radial  sine  surface  of  1-igurc  2.  Indeed, 
if  the  albedo  is  roughly  constant,  the  form  of  the  equations  indicates  that  there  may  be  topog.rapbic 
effects  that  could  also  cause  shaip  changes  in  irradiancc.  I  hc  surface  conditions  under  which  such  a 
change  in  irradiancc  can  iKcur  arc  important,  since  tlic  absence  of  zero-crossings  in  a  region  would 
then  imply  the  absence  of  such  surface  conditions  for  that  region.  It  is  precisely  these  restrictions 
on  surface  shape  which  will  allow  us  to  determine  a  surface  consistent  with  the  depth  values  along 
the  zero-crossing  contours.  I  hc  basic  problem  is,  under  what  conditions  docs  bending  of  die  surfice 
force  an  inllcction  in  the  irradiancc  array?  I'his  question  will  be  answered  in  the  following  sections  by 
considering  specific  eases. 

I'hcrc  .ire  two  points  to  note,  before  beginning  the  mathematical  details.  Ibc  first  concerns  tlie 
. i«  o.vosinii'tino  ibp  nriinal  ski  icli  rcnroscnialions  unmi  which  the  stereo 
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the  visual  system  must  incorporate  some  additional  constraints  besides 
the  known  points  in  order  to  compute  the  complete  surface. 

Using  the  image  irradiance  equation,  we  derive  the  surface  consistency 
constraint,  informally  referred  to  as  no  news  is  good  news.  The 
constraint  Implies  that  the  surface  must  agree  with  the  information 
from  stereo  or  motion  correspondence,  and  not  vary  radically  between 
these  points.  An  explicit  form  of  this  surface  consistency  constraint 
is  derived,  by  relating  the  probability  of  a  zero-crossing  in  a  region 
of  the  image  to  the  variation  in  the  local  surface  orientation  of  the 
surface,  provided  that  the  surface  albedo  and  the  illumination  are 
roughly  constant. 

The  surface  consistency  constraint  can  be  used  to  derive  an  algorithm 
for  reconstructing  the  surface  that  "best"  fits  the  surface  information 
provided  by  stereo  or  motion  correspondence. 
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ARS  I RACTT  Computational  theories  of  staicturc-from-motion  and  stereo  vision  only  specify 
tlic  compuliilion  of  tlircc-dimcnsional  surface  infonnatiun  at  points  in  the  image  at  which  the  ir- 
radiance  changes^'— Tor  example,  the  zero-crosifngs  of  a  operator  applied  to  the  image/  Yet,  the 
visual  perception  is  clearly  of  complete  surfaces,  and  this  perception  is  ci'nsistcnt  for  different  observ¬ 
ers.  Since  mathematically  tlie  class  of  surfaces  which  could  pass  tlirough  the  known  boundary  points 
provided  hy  the  sicre<»  system  is  infinite  and  contains  widely  varying  surfaces,  the  visual  system  must 
incorporate  some  additional  constraints  besides  the  known  points  in  order  to  compute  die  complete 
surface. 


Using  the  image  irradiance  equation,  we  derive  the  surface  consistency  constraint,  informally 
referred  to  as  no  news  is  good  news.  Tlic  constraint  implies  that  the  surface  must  agree  witli  the 
information  from  stereo  or  motion  correspondence,  and  not  vary  radically  between  these  points.  An 
explicit  form  of  this  surface  consistency  constraint  is  derived,  by  relating  the  probability  of  a  zero¬ 
crossing  in  a  region  of  the  image  to  die  variation  in  the  local  surface  orientation  of  the  surface, 
provided  that  the  surface  albedo  and  the  illumination  arc  roughly  constant. 

The  surface  consistency  constraint  can  be  used  to  derive  an  algorithm  ^  reconstructing  the 
surface  that  “best”  fits  the  surface  information  provided  by  stereo  or  motion  correspondence. 


This  report  describes  research  done  at  the  Artificial  Intelligence  Laboratory  of  the  Massachusetts 
Institute  of  Technology.  Support  for  the  laboratory's  artificial  intelligence  research  is  provided  in 
part  by  the  Advanced  Research  Projects  .Agency  of  the  Department  of  Defense  under.  Office  of 
Naval  Research  contract  N00014-80'C-0505  and  in  part  by  National  Science  Foundation  Grant  79- 
23110MCS. 


1.  Introduction 


Although  our  world  has  Uircc  spatial  dimensions,  the  projection  of  light  rays  onto  the  retina 
presents  our  visual  system  with  an  image  of  tlie  world  tliat  is  inherently  two-dimensional.  We  must 
use  such  images  to  |)hysically  interact  with  tltis  ihiee-dimcnsional  world,  even  in  situations  new  to 
us.  or  with  objects  unknown  to  us.  That  we  easily  do  so  implies  tliat  one  of  toe  functions  of  the 
human  visual  system  is  to  reconstruct  a  tlirec-diinensional  representation  of  the  world  from  its  two- 
dimensional  projection  onto  our  eyes. 

Methods  tliat  could  he  used  to  effect  tliis  thrcc-diincnsional  reconstruction  include  stereo  vision 
[e.g.  Wheatstone,  1838;  Helmholtz,  1925;  Jules/.,  19?ljaiid  structure  from  motion  (for  example.  Miles, 
1931;  Wallach  and  O'Connell,  1953;  Johannson.  1964).  Both  of  tlicse  methods  may  be  considered  as 
correspondence  techniques,  since  tiiey  rely  on  esmblishing  a  correspondence  between  identical  items 
in  different  images,  and  using  the  difference  in  projection  of  these  items  to  determine  surface  shape. 

Most  of  tlie  current  coinputational  tlieories  of  these  prtx;csses  jMarr  and  Poggio,  1979;  Crimson, 
1980,  1981a;  Mayhew  and  Frisby.  1981;  Ulfman.  1979;  Longuct- Higgins  and  Prazdny,  1980}  argue 
tliat  the  correspondence  prcKCSS  cannot  take  place  at  all  points  in  an  image.  Rather,  the  first  suige 
of  the  correspondence  process  is  to  derive  a  symbolic  description  of  points  in  the  image  at  which  the 
irradiance  undergoes  a  significant  change  [Marr  and  Hildretli,  1980].  Ihis  symbolic  representation 
(called  the  primal  sketch  (Marr,  1976;  Marr  and  Hildreth,  1980])  forms  the  input  to  tlie  second  stage 
of  the  process  in  which  the  actual  correspondence  is  computed.  As  a  consequence  of  the  form  of  the 
input,  the  correspondence  priKess  can  compute  explicit  surface  information  only  at  scattered  points 
in  the  image.  Yet  our  perception  is  clearly  of  complete  surfaces.  For  example,  in  Figure  1,  a  sparse 
random  dot  stereogram  yields  the  vivid  perception  of  a  square  floating  in  space  above  a  background 
plane,  rather  than  a  collection  of  dots  suspended  in  space.  Ihis  suggests  that  some  type  of  filling-in,  or 
interpolation  of  surface  information,  is  taking  place  in  the  visual  system. 

7his  poses  an  interesting  problem,  since  mutheinatically  the  class  of  surfaces  that  could  pass 
through  the  known  boundary  points  provided  by  the  stereo  algorithm  is  infinite  and  contains  widely 
varying  surfaces.  The  implication  of  our  perception  of  complete  surfaces,  which  is  consistent  for 
different  viewers,  is  that  the  visual  system  must  incorporate  some  additional  constraints,  besides  the 
known  points,  in  order  to  compute  the  complete  surface. 
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Figure  1.  A  Sparse  Random  Dot  Pattern.  Although  the  density  of  dots  is  very  sni:ill.  Die  perception 
obtained  tipun  fusing  this  pattern  is  one  of  two  disjoint  planes,  rather  than  dots  isolated  in  (^th. 


In  this  paper,  we  will  use  the  image  irradiance  equation  (Horn,  1977]  to  derive  the  surface 
coiisisicncy  conslraiiit,  informally  referred  to  as  no  news  is  good  news.  The  constraint  implies  that  the 
surfiicc  must  agree  with  the  information  from  stereo  or  motion  correspondence,  and  not  vary  radically 
between  these  points.  We  will  derive  an  explicit  form  of  this  surface  consistency  constraint,  by  relating 
the  probability  of  an  abrupt  irradiance  change  in  a  region  of  the  image  to  the  variation  in  the  local 
surface  orientation  of  the  corresponding  surface.  We  will  then  indicate  how  this  constraint  can  be 
used  to  derive  an  algorithm  for  reconstructing  the  surface  that  “best”  fits  the  surface  information 
provided  by  stereo  or  motion  correspondence  (Grimson.  1981b,  1982], 


2.  The  Computational  Constraint 

The  input  representation  used  by  the  stereo  algorithm  [Marr  and  Poggio,  1979;  Grimson,  1981a, 
198 lb]  consists  of  a  symbolic  representation  of  points  at  which  the  image  irradiance  undergoes  a 
significant  cliange.  Ihcsc  points  arc  identified  by  the  zero-crossings  of  a  Laplacian  of  a  Gaussian 
(V^G)  operator  applied  to  the  image;  that  is,  the  image  is  convolved  with  a  V^G  operator  and  the 
points  at  which  the  resulting  convolution  changes  sign  arc  luciited  [Marr  and  Hildreth,  1980;  Hildreth, 
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1980],  As  a  consequence  of  the  correspondence  process,  explicit  three-dimensional  infonnation  about 
the  surface  shape  will  be  computed  only  at  these  /ero-crj>ssings. 

Suppose  one  were  to  attempt  to  construct  a  complete  surface  description  based  only  on  the 
surhice  information  known  along  tlie  /ero-crossings  (where  by  complete,  we  mean  tliat  an  explicit 
surface  value  is  assigned  to  each  point  on  a  rectiingular  grid).  An  infinite  number  of  surfaces  would 
consistently  fit  tlie  boundary  conditions  provided  by  these  surfaee  values.  Yet  there  must  be  some 
way  of  deciding  which  surface,  or  at  least  which  small  family  of  surfaces,  eould  give  rise  to  f.ie  zero- 
crossing  descriptions.  Ihis  means  tliat  tliere  must  be  some  additional  information  available  from  the 
visual  process  which,  when  taken  into  account,  will  identify  a  class  of  nearly  indistinguishable  surfaces 
tliat  represent  the  visible  surfaces  of  a  scene. 

In  order  to  determine  what  infonnation  is  available  from  the  visual  system,  one  must  first  care¬ 
fully  consider  the  prrKess  by  which  tlie  zero-crossing  contours  arc  generated.  For  insuincc,  sudden 
changes  in  die  reflectance  of  a  surface,  caused  by  surface  scratches  or  texture  markings,  will  give  rise 
to  zero-crossings  in  the  convolved  image  [Mart  and  Hildreth.  1980;  Hildreth.  1980).  Of  more  interest 
here  is  die  fact  diat  sudden  or  sharp  changes  in  the  shape  of  die  surface  will  under  must  ciicumstanccs 
also  give  rise  to  zero-crossings.  Iliis  can  be  used  to  constrain  the  possible  shapes  of  surfaces  that  could 
pn'ducc  particular  surface  values  along  the  zero-crossing  contours. 

We  illustrate  the  basic  argument  with  an  example.  Suppose  one  is  given  a  closed  zero-crossing 
contour,  within  which  there  arc  no  other  zero-crossings.  An  example  would  be  a  circular  contour, 
along  which  the  depth  is  constant.  There  arc  many  surfaces  which  could  fit  this  set  of  boundary 
conditions.  One  such  surface  is  a  flat  disk.  One  could,  however,  also  fit  other  smooth  surfaces  to  this 
set  of  boundary  conditions.  For  example,  the  highly  convoluted  surface  formed  by  8in(**  -|- 
would  be  consistent  with  the  known  disparity  values  (see  Figure  2).  Yet  in  principle,  such  a  rapidly 
varying  surface  should  give  rise  to  other  zero-crossings.  This  follows  from  the  observation  Uiat  if  the 
surface  orientation  undergoes  a  periodic  variation,  then  it  is  likely  that  the  irradiance  values  will  also 
undergo  such  a  variation.  Since  the  only  zero-crossings  lie  at  the  borders  of  the  object,  this  implies 
that  the  surface  sin  (x^  is  a  highly  improbable  representative  surface  for  this  set  of  boundary 

conditions. 

Hence,  the  hypothesis,  which  will  be  evaluated  in  the  following  sections,  is  that  the  set  of  zero¬ 
crossing  contours  contains  implicit  information  about  the  surface  as  well  as  explicit  information.  If 
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Figure  2.  Possible  Surfaces  Fitting  Depth  Values  at  Zero-Crossings.  Given  boundiir)’  condHiont 
of  a  circului  zero-crossing  contour,  along  which  the  depth  is  constant,  there  are  many  posnble 
surfaces  which  could  fit  the  known  depth  points.  Two  examples  are  a  flat  disk,  and  the  highly 

convoluted  surface  formed  by  sin  (x’  -f-  y^)^.  shown  here.  (From  Grimsoa  11981b).) 

unc  can  determine  a  set  of  conditions  on  the  surfiicc  shtqx  that  cause  inflections  in  the  irradiance 
values,  then  one  may  be  able  to  determine  a  likely  surface  structure,  given  a  set  of  boundary  condi¬ 
tions  along  the  zero-crossing  contours. 

2.1  No  News  is  Good  News 

In  general,  any  one  of  a  multitude  of  widely  varying  surfaces  could  fit  the  boundary  conditions 
imposed  along  the  zero-crossings.  Our  intention  is  to  show  that  to  be  completely  consistent  with 
the  imaging  process,  such  surfaces  must  meet  both  explicit  cunditions  and  implkil  conditions,  llie 
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explicit  conditions  arc  given  by  tlic  depth  values  along  the  zero-crossing  contours.  Ilic  implicit  condi¬ 
tions  arc  that  the  surface  must  not  impose  my  zero-crossing  contours  other  than  those  which  appear 
in  tlic  convolved  image.  This  implicit  condition  leads  to  the  surface  cunsisiciicy  amsiniini  [Grimson, 
1981b],  namely: 

'I'hc  absence  of  /eru-crossiugs  const rain.s  the  possible  surface  sbape.s. 

Just  as  the  presence  of  a  zero-crossing  tells  us  that  some  physical  property  is  changing  at  a  given 
location,  tlic  absence  of  a  zero-crossing  tells  us  the  opposite,  that  no  physical  property  is  changing, 
and  in  particular  that  the  surface  topography  is  not  changing  in  a  radical  manner.  We  informally 
refer  to  lliis  constraint  as  no  news  is  good  news,  since  it  says  that  in  general  tlic  surface  cannot  change 
radically  without  infonning  us  of  this  fact  by  means  of  zero-crossings.  Note  that  in  practice,  the 
convolved  image  may  not  signal  the  entire  extent  of  a  discontinuity  in  tlte  surface  topography,  but  at 
least  some  portions  of  the  discontinuity  will  be  evidenced  by  a  zero-crossing.  In  order  to  complete 
such  a  partial  discontinuity,  a  subjective  contour  must  be  inserted  between  die  known  portions  (Urady 
and  Grimson,  1981]. 

In  order  to  make  explicit  any  constraints  on  die  shape  of  the  surface  for  locations  in  the  image 
not  associated  with  a  zero-crossing,  we  will  carefully  examine  the  factors  which  influence  irradiance 
and  hence  image  intensity.  (Note  dial  while  irradiance  refers  to  die  illuminant  flux  emitted  from  the 
surface  of  an  object  and  intensity  refers  to  die  brightness  recorded  by  a  sensor,  we  will  use  the  terms 
intcrchangably,  since  they  are  proportional  [Horn  and  Sjoberg,  1979].)  These  factors  will  be  expressed 
in  die  image  irradiance  equation,  which  describes  the  manner  in  which  a  particular  irradiance  is 
formed  at  a  point  in  the  image.  Two  goals  are  kept  in  mind.  The  first  is  to  determine  what  surface 
conditions  will  cause  a  local  change  in  irradiance,  and  the  second  is  to  combine  this  constraint  with 
the  input  from  the  visual  proccs.scs,  .such  as  stereo  or  structurc-from-motior;.  in  order  to  design  an 
algoridim  for  interpolating  surface  information. 

I’hc  basic  result,  corresponding  to  die  intuitive  argument  given  above,  is  that  the  probability  of  a 
zero-crossing  occurring,  in  regions  where  the  illumination  is  roughly  constant  and  the  surface  material 
docs  not  change,  is  a  monotonic  function  of  the  variation  in  the  orientation  of  the  surface  normal. 
This  means  that  the  probability  of  a  zero-crossing  increases  as  the  variation  in  surface  orientation 
increases.  By  inverting  this  argument,  we  will  show  that  the  best  surface  to  fit  the  known  data  is  that 
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which  minimizes  the  variation  in  surface  orientation  since  this  surface  is  most  consistent  with  die  zero- 
crossings  in  tlic  convolved  image. 


3.  Image  Formation 

In  order  to  examine  die  prtKCSs  of  zero-crossing  formation,  a  review  of  die  factors  involved  in 
the  form  -  ion  of  an  image  will  he  presented.  One  factor  is  die  geometry  of  die  projection  (either 
orthographic  or  perspective)  frem  die  scene  to  the  image.  The  second  factor,  wliich  we  will  review 
briefly  below,  is  die  manner  in  wliich  image  irradiance  values  at  a  point  are  formed.  A  detailed 
analysis  of  diesc  processes  has  been  undertaken  by  several  investigators  |Horn.  1970,  1975,  1977; 
Nicodemus,  ct  al..  1977;  Woodham.  1978;  Horn  and  Sjoberg,  1979]. 

3.1  Grey-Level  Formation 

In  this  section,  we  will  outline  die  faettns  involved  in  the  creation  of  image  irradiances  in  order 
to  illustrate  the  relationship  between  chtingcs  in  irradiance  and  zero-crossings  in  the  convolved  image, 
(most  of  the  discussion  is  based  on  Horn  and  Sjoberg,  fo  do  this,  we  need  to  determine  what 

inadiance  value  will  be  associated  with  a  particular  image  liKation. 

Ilic  apparent  “brightness”  recorded  by  an  imaging  device  is  a  nicasurcment  of  image  irradiance 
E,  the  radiant  flux  striking  a  unit  area  of  the  receptive  field.  Ihc  flux  reaching  a  small  portion  of 
the  receptive  field  will  be  exclusively  a  function  of  a  corresponding  small  surface  element  on  some 
object  in  the  scene,  provided  the  imaging  system  is  properly  fiKuscd  and  the  lens  is  small  relative  to 
its  distance  from  the  object  IHorn  and  Sjoberg,  1979].  We  want  to  specify  tlic  fiictors  tliat  determine 
ihc  image  irradiance  at  a  point  in  the  image,  as  a  function  of  the  corrc.sponding  surface  elemenL 

In  general,  there  arc  ilircc  factors  governing  tlic  irradiance  recorded  at  any  position  in  the 
image: 

(1)  the  amount  of  radiant  flux  striking  a  surface,  a  rtinctioti  of  the  distance  of  the  surface  from  the 
source  r,  the  orientation  of  the  surface  relative  to  the  source,  and  the  intensity  /  of  the  source 
itself; 

(2)  the  percentage  of  incident  flux  reflected  by  the  surface  (as  opposed  to  the  percentage  absorbed 
or  transmiltcd),  a  fiinction  of  the  surface  material  and  usually  described  bv  the  albedo  p'.  and 
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(3)  the  distribution  of  that  reflected  flux  as  a  function  of  direction,  usually  dcscrilwd  by  a  reflectance 

fttnetion. 

We  shall  show  Uiat  a  sharp  change  in  each  of  tliese  factors  will  give  rise  to  a  zero-crossing  in  die  con¬ 
volved  image.  Further,  in  regions  where  both  die  incident  illumination  and  abstirption  characteristics 
of  the  suifacc  material  are  roughly  constant,  an  equation  relating  Uic  recorded  irradianccs  to  the  shape 
of  die  surface  will  be  derived.  This  image  irradiance  equation  will  be  used  in  later  sections  to  verify 
our  surfitcc  consistency  constraint. 

Most  surfaces  have  the  property  diat  the  reflectance  is  not  changed  by  rotating  a  surface  element 
about  its  nonnal  (exceptions  are  diffraction  gratings,  irridescent  plumage  and  “tiger’s  eye”).  Such 
surfaces  arc  referred  to  as  isotropic.  If  a  surface  possesses  this  property,  only  three  angles  arc  needed 
to  dctcnninc  reflectance,  as  shown  in  F-igurc  3.  Hie  angle  between  the  local  surface  normal  and  the 
incident  ray  is  called  the  incident  angle  and  is  denoted  by  i.  The  angle  between  the  local  surface 
normal  and  the  emitted  ray  is  called  die  view  angle  and  is  denoted  by  e.  The  angle  between  the 
incident  and  emitted  rays  is  called  the  phase  angle  and  is  denoted  by  g.  In  the  following,  we  shall 
assume  that  the  surfaces  arc  isotropic.  This  reduces  the  image  irradiance  to  a  fiinction  of  /,  r,  t,  e,  g 
and  p.  where  /  is  the  intensity  of  the  source,  r  is  the  distance  of  the  source  from  die  surface,  t,  e,  and  g 
arc  the  angles  specified  above,  and  p  is  die  albedo  of  die  surface  material. 

We  know  that  the  incident  flux  density  follows  an  inverse-square  reduction  as  a  function  of  dis¬ 
tance.  If  two  surfaces  lie  at  very  different  depths,  and  all  other  factors  arc  roughly  equal,  this  inverse- 
square  dependence  will  cause  a  noticeable  difference  in  the  irradianccs  associated  with  the  different 
surfaces.  If  die  projections  of  the  two  surfaces  arc  adjacent  in  die  image,  the  different  irradianccs 
will  give  rise  to  a  zero-crossing  in  the  convolved  image.  This  is  to  be  expected,  since  the  Marr- 
Hildreth  theory  of  edge  detection  was  based  on  the  requirement  of  detecting  changes  in  the  image 
corresponding  to  changes  in  some  physical  property  of  the  surfaces.  In  this  case,  the  changing  physical 
properly  is  object  continuity. 

Given  that  these  situations  will  cause  a  zero-crossing,  we  can  restrict  our  attention  to  regions  of 
the  image  between  the  borders  of  the  objects.  We  will  further  assume  that  the  source  is  distant  relative 
to  the  surfaces.  This  has  two  consequences.  The  first  is  that  each  surface  receives  roughly  uniform 
illuminadon.  This  implies  that  the  //r*  term  is  roughly  constant  and  may  be  ignored  in  what  follows. 
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F  igure  3.  The  Imaging  Geomciry.  The  incident  angle  t  is  the  angle  between  the  incident  ray 
and  surface  normal.  Ihe  view  angle  e  is  the  angle  between  the  emergent  ray  and  the  surface 
normal  The  phase  angle  g  is  the  angle  between  the  incident  and  emergent  rays.  (Rcdntwn  from 
Woodhiuit.  (1978IT 


1110  second  is  that  the  angle  between  tJic  viewe-  direction  and  the  source  direction  (the  phase  angle  g) 
will  be  roughly  constant  over  each  surface. 

If  the  surface  material  changes,  there  will  frequently  be  an  asstKiated  change  in  the  albedo.  If 
dll  other  factors  arc  roughly  equal,  tliis  will  cause  a  change  in  the  irradianccs  and  there  will  be  a 
corresponding  rcro-crossing  in  tlic  cotivolved  image.  This  is  again  expected  from  the  Marr-Hildrcth 
theory  of  edge  detection;  a  changing  physical  property  of  the  surface  should  correspond  to  a  zero- 
crossing  in  the  convolved  image.  In  general  we  shall  restrict  our  attention  to  regions  of  the  surfaces 
between  such  material  changes,  so  that  the  albedo  p  is  roughly  constant.  In  these  regions,  we  can 
consider  the  image  irradianccs  to  be  a  function  of  t,  e  and  p  (ignoring  scale  constants). 

If  we  assume  that  the  source  is  distant  enough  and  small  enough  to  be  treated  as  a  point  source, 
then  we  can  simplify  the  geometry  further  by  reducing  the  angles  t  and  e  to  a  specification  of  the  local 
surface  normal,  in  the  following  manner.  Consider  a  Cartesian  coordinate  system  with  a-axis  aligned 
with  the  line  of  sight,  and  z-axis  some  arbitrary  direction  in  the  plane  normal  to  the  line  of  sight. 
Suppose  tire  surface  is  specified  in  UiC  coordinate  system  by  z  =  /(z,  y).  Elementary  calculus  states 
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that  the  vectors 


{■'“•S}  {<} 

arc  tangent  to  llie  surface  tit  a  given  point.  The  cross  product 

(dz  dz  ) 
Xdx'^j'  ] 

is  normal  to  die  tangent  plane,  and  lienee  to  the  surface.  IX'fining 


P  = 


dz 

dx 


<1 


dz 


the  normal  vector  becomes  {p,  q,  —  1}.  I  hc  quantity  {p,q)  is  usually  referred  to  as  the  gradient  and 
the  space  of  all  points  (p,  q)  is  referred  to  as  gradient  si>acc\HuKnMm.  1971;  Mackworth,  1973]. 

In  this  case,  the  angles  i  and  e  can  be  straightforwtirdly  tninsformcd  into  die  normal  components 
p  and  q  (Horn,  1977].  Since  we  arc  considering  the  case  of  a  disuint  point  source,  its  direction  rcladve 
to  die  surface  can  be  represented  by  a  direction  vector  {p,„  g,„  —  1}.  A  siraightfoward  calculation  of 
die  vector  dot  products  shows  dial  die  angles  t  and  e  are  given  by: 


cos(t)  = 
cos(e)  = 


1  +  m  + 


-1-  9'Vl  +  P'i  +  9? 

1 


v/l+p2+g2’ 


Extended  sources  (sources  whose  si/.e  relative  to  their  separadon  from  the  scene  cannot  be  treated 
as  a  point  source,  for  example,  fluorescent  lights  in  a  normal  room)  can  be  frequently  treated  by  su¬ 
perposition  of  point  sources,  or  as  being  equivalent  to  an  appropriately  chosen  point  source  [Brooks, 
1978;  Silver,  1980].  Tlius,  the  irradiance  reduces  to  a  ftinction  ofp,  q  and  p. 

Finally,  if  we  assume  diat  the  objects  arc  distant  relative  to  the  viewer,  the  image  projection  may 
be  treated  as  orthographic.  In  this  case,  the  coordinate  system  of  the  image  and  the  coordinate  system 
of  the  scene  can  be  treated  as  idcndcal  so  that  we  obtain  the  image  irradiance  equation: 


E{x,  y)  =  p{x,  y)R{p[x,  y),  q[z,  y)) 


where  E{x,  y)  is  the  image  irradiance  recorded  at  a  point  in  the  image,  p  is  the  albedo  associated 
with  die  surface  intersecting  the  ray  from  (x,  y)  and  R{p,  q)  is  die  reflectance  map  (Horn,  1970,  1975, 
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and  Sjobcrg,  1979,  Horn  and  Bachman,  1977],  Note  that  the  reflectance  map  72  difTcrs  from 
flectance  function  for  a  surface  (Nicodemus,  et  al..  1977].  Ihc  reflccl.inco  function  describes 
nount  of  flux  reflected  in  a  particular  direction,  given  an  incident  flux  from  a  second  given 
on.  Tlie  reflectance  map  coinbimis  tlie  effects  oT  the  roflcxtancc  function  in  the  ease  of  uniform 
nation  and  isotropic  surface  material  with  the  illumination  geometry  and  the  image  sensor’s 
point.  In  the  case  of  constaiv,  albedo  the  image  irradiance  cq’iation  reduces  to  the  partial 
eiv  ■’  equation  (Horn.  1970.  1975:  Horn  and  Sjoberg,  1979J; 

E(x,  y)  =  R{  p{x,  y),  q{x,  y) ). 

two  equations  describe  tlie  manner  in  which  an  Image  irradiance  (or  grey-level)  is  obtained  at  a 
ular  point  in  tlie  image. 

4.  The  Interpolation  Theorem 

We  haie  seen  that  zero-crossings  can  arise  from  many  factors.  We  shall  restrict  our  attention  to 
ins  in  which  illumination  is  constant,  albedo  is  roughly  constant  and  surface  material  is  isotropic, 
inicntioi  is  to  .show  that  in  such  eases,  if  the  surface  topography  changes  nidically,  the  image 
ianccs  must  also  change  radically.  We  will  then  be  able  to  use  tlie  contrapositivc  statement —  if 
mage  irr  idianccs  do  not  change  radically,  tlicn  the  surface  topography  also  docs  not  change.  Note 
if  die  albedo  or  die  illumination  is  not  coriStant  over  die  region,  we  cannot  make  this  statement 
is.  there  could  be  situations  in  which  the  surface  topography  docs  change  radically  without 
ng  a  corresponding  change  in  the  image  irradianccs.  because  one  of  the  other  factors,  such  as 
Ihedo.  is  also  changing  in  such  a  manner  as  to  mask  out  die  effects  of  the  change  in  topography, 
c  situations  arc  fortunately  very  rare,  since  they  require  a  precise  meshing  of  the  effects  of  the 
ging  albedo  with  the  changing  surface  topography  and  in  general  arc  highly  dependent  on  the 
cr  ptsitijii  and  illuminant  direction.  7hus.  a  small  change  in  the  viewer  position  will  usually 
uple  die  conflicting  effects  of  albedo  and  surface  shape,  and  the  ch.mgc  in  topography  will  give 
o  a  zero-crossing.  (In  other  words,  it  is  generally  difficult  to  paint  a  radically  curved  surface  so 
appear  flat  from  all  viewpoints.  'Hie  opposite  is  not  true,  of  course.  One  can  easily  paint  a 
III  face  to  appear  curved.  In  other  words,  we  prefer  to  see  topographic  changes,  as  evidenced  by 
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the  infomial  observation  that  photograplis  with  smoothly  changing  albedo  are  usually  interpreted  as 
having  smoothly  changing  topography. ) 

1110  concern  here  is  whether  there  can  be  zero-crossings  corresponding  to  topographic  rather 
tlian  photometric  changes  in  an  object,  for  example,  in  the  radial  sine  surface  of  1-igurc  2.  Indeed, 
if  dic  albedo  is  rouglily  constant,  tlic  ftnm  of  die  equations  indicates  that  there  may  be  topographic 
elTccts  that  could  also  cause  shaip  changes  in  irradiancc.  I'hc  surHicc  conditions  under  which  such  a 
change  in  irradiancc  can  cKcur  arc  important,  since  die  absence  of  zero-crossings  in  a  region  would 
then  imply  the  absence  of  such  surface  conditions  for  that  region.  It  is  precisely  these  restrictions 
on  surface  shape  which  will  allow  us  to  detennine  a  surface  consistent  with  die  depth  values  along 
the  zero-crossing  contours.  The  basic  problem  is,  under  what  conditions  docs  bending  of  die  surface 
force  an  inilcction  in  the  irradiancc  array?  'fhis  question  will  be  answered  in  the  following  sections  by 
considering  specific  cases. 

There  arc  two  points  to  note,  before  beginning  the  madieniatical  details.  I'lic  fiist  concerns  die 
role  of  the  Gaussian  filter  in  constmeting  the  primal  sketch  representations  upon  which  the  stereo 
correspondence  is  computed.  The  Gaussian  pcrfoims  a  smoothing  of  die  image,  theieby  isolating  ir- 
radiance  changes  at  a  particular  scale.  At  die  same  time,  it  removes  some  of  the  noise  problems  which 
arise  from  using  a  discrete  grid  to  represent  die  image.  In  what  follows,  we  shall  concentrate  on  the 
difTcrcnlial  operator  ( V'^)  and  its  role  in  the  creation  of  die  zero-crossings.  The  effect  of  the  Gaussian 
will  not  be  considered  in  any  detail.  Hence,  die  surface  reconstruction  will  not  account  for  minor 
surface  fluctuations  on  the  scale  of  the  grid  resolution.  I'his  is  not  a  major  problem.  (I-lquivalcntly, 
the  problem  may  be  considered  as  one  of  applying  die  I.aplacian  to  the  image  G  *  E.  ITie 
reconstruction  will  be  based  on  this  image  information.) 

The  second  point  concerns  the  use  of  the  Lnplacian  operator  as  opposed  to  directional 
derivatives.  The  mathematical  arguments  which  follow  arc  based  on  the  consideration  of  zero- 
crossings  of  the  convolution  of  a  directional  second-order  dificrcntial  operator  with  die  image,  (»  • 
V)  *  E.  We  have  already  seen  that  the  Marr-Hildrcth  theory  of  edge  detection  is  based  on  the  use 
of  zero-crossings  of  the  image  convolved  with  Uie  I.aplacian,  V'^  *  E.  This  difference  is  not  critical, 
for  the  following  two  reasons.  Marr  and  Hildreth  show  that  under  some  simple  assumptions,  the 
zero-crossings  obtained  by  either  operator  are  identical.  In  particular,  the  condition  of  linear  variation 
(Marr  and  Hildreth,  1980]  states  that:  the  irradiancc  variation  near  and  parallel  to  a  line  of  zero- 
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crossings  should  locally  be  linear.  If  tills  condition  is  assumed  to  be  true  then  it  can  be  shown  (Marr 
and  Hildreth,  1980.  Appendix  A]  that  the  zero-crossings  obtained  with  the  haplacian  arc  identical  to 
those  obtained  using  directional  derivatives.  Thus,  in  tiiose  situations  tor  which  the  condition  of  linear 
variation  is  valid,  no  difference  is  obtained.  In  the  ease  where  the  condition  of  linear  variation  docs 
not  hold,  the  difference  in  the  zero-crossings  of  tlie  operators  lies  in  their  position,  not  in  tlic  existence 
of  such  zero-crossings.  For  example,  at  a  corner  (which  frequently  carries  strong  visual  information), 
both  V-  '  E  and  (v  •  V)  *  £  will  give  rise  to  zero-crossings.  The  only  difference  will  be  in  tlic  exact 
position  of  tlic  /cro-cros.sirigs.  We  shall  see  that  such  variation  in  position  is  of  negligible  consequence 
to  tlic  reconstruction  process.  Hence,  it  will  be  assumed  that  the  arguments  developed  in  the  next 
section,  based  on  the  zero-crossings  of  a  directional  second  derivative  will  also  generally  apply  to  the 
zero-crossings  of  the  (.aplacian. 

Finally,  in  what  follows,  tlic  functions  ,0,  R  and  /  arc  assumed  to  have  continuous  second  order 
partial  derivatives.  Throughout,  it  is  assumed  that  the  albedo  effects  can  be  ignored  relative  to  the 
topographic  ones,  tliat  is.  the  albedo  factor  may  be  considered  roughly  consunt  and  will  therefore  not 
afl'cet  any  study  of  derivatives.  Without  loss  of  generality,  one  may  assume  tliat  p  =  1  and  ignore  the 
albedo  in  what  follows. 

4.1  General  Argument 

The  basic  hypothesis  is  that  in  order  for  a  surface  to  be  consistent  with  a  given  set  of  zero- 
crossings,  not  only  must  it  give  rise  to  a  zero-crossing  in  the  convolved  irradianccs  at  those  points, 
but  it  must  also  not  give  rise  to  zero-crossings  anywhere  else.  Under  most  situations,  this  restriction 
would  require  that  the  surface  not  change  in  a  radical  manner  between  zero-crossing  contours.  (For 
example,  the  surface  shown  in  Figure  2  is  not  consistent  with  the  boundary  conditions  of  a  circular 
zero-crossing  contour  of  uniform  depth).  It  is  difficult  to  prove  this  assertion  in  general,  since  the 
image  forma  ion  equation  includes  terms  dependent  on  the  imaging  process  and  on  the  light  source 
geometry',  as  well  as  factors  dependent  on  the  photometric  properties  of  the  surface  itself.  However, 
under  some  l  airly  weak  assumptions  concerning  the  relative  strengths  of  photometric  and  topographic 
changes,  and  the  form  of  tlic  reflectance  function,  we  will  prove  the  surface  consistency  theorem, 
describing  the  probability  of  a  zero-crossing  as  a  function  of  the  shape  of  the  surface.  The  importance 
of  this  theorem  is  that  it  leads  to  a  method  for  measuring  the  probability  of  a  particular  surface  being 
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Figure  4.  An  Example  of  a  Developable  Surface.  The  component  of  the  surface  orientation  in 
the  y  direction  is  constant  for  this  region  of  the  surface,  sc'  that  the  only  variations  in  surface 
orieiilatioii  lake  place  in  the  x  diteciion. 


inconsistent  with  the  zero-crossing  infonnalion.  ITtis  in  turn  suggests  that  it  will  be  possible  to  derive 
a  method  for  determining  the  best  possible  surface  to  fit  tlic  known  information  [Crimson,  1982], 

4.2  A  One-Dimensional  Example 

To  illustrate  the  scope  of  the  surface  consistency  theorem,  we  shall  consider  first  the  one- 
dimensional  case  of  a  developable  surface,  before  proving  the  general  theorem  concerning  arbitrary 
surfaces.  Note  that  the  Laplacian  is  orientation  independent,  so  that  without  loss  of  generality,  one 
may  rotate  the  coordinate  system  of  the  image  to  suit  our  needs.  One  may  assume  that  the  surface  has 
the  form  f(x,  y)  such  that  /„(*,  y)  =  q(x,  y)  =  c,  in  the  local  region  under  consideration.  Hence, 
the  partial  derivatives  of  q  vanish,  as  do  any  partial  derivatives  of  p  involving  y.  A  sample  surface  is 
shown  in  Figure  4. 

Suppose  that  a  one-dimensional  slice  in  the  i  direction  of  the  surface  contains  at  least  two 
inflection  points.  Figure  S  indicates  a  sample  surface  and  its  derivatives.  Since  q  is  assumed  constant, 
the  derivatives  of  the  image  irradiantc  equation  are  given  by: 
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V^E  =  +  2p^pPx  -f  f^RppPl  +  RpPz^y 

(Note  that  except  when  explicitly  stated  otherwise,  we  shall  use  a  subscript  to  denote  a  partial  deriva¬ 
tive,  so  that,  for  example,  Px  =  dpjdx.)  In  tliis  situation  of  a  developable  surface,  we  can  prove  the 
following  result 

Theorem  0:  Consider  a  portion  of  a  second  differentiable,  developable  surface  oriented  along  the  y 
axis  SUCH  ,hatffj{x,y)  =  q  =  e,  for  some  constant  c.  If  the  following  conditions  are  true: 

1.  The  surface  portion  contains  exactly  two  inflection  points  in  ihex  direction,  atxi  andX2, 

2.  At  the  points  Xi  and  ij,  normalized  changes  in  albedo  are  dominated  by  normalized  changes  in 
reflectance, 

V^p  ^  RpPxx 

'T  ^  R  ' 

3.  The  reflectance  map  R  does  not  pass  through  an  extremum  in  this  region  of  the  surface, 

4.  The  reflectance  map  R  is  not  constant  over  this  region  of  the  surface. 

5.  The  albedo  p  is  non-zero, 

then  there  ex  ists  a  point  <  *'  <  *2  such  that  V*E(arQ  =  0.  That  is,  there  exists  a  zero-crossing 
here. 

Proof,  ilic  signum  function  is  defined  by: 

{1  if  I  >  0 

—1  if  *  <  0 

0  if  *  =  0 

From  the  derivatives  of  the  image  equation, 

V^E  =  {V^p)R  +  2pxRpP.  +  dRppf^x  +  RpPxr) 

At  the  inflection  points  X|  and  X2>  pAtt^  ~  0.  Hence,  evaluation  of  the  equation  yields 

V*£:(*0  =  +  p{*i)Hp{p(*.))p,x(*i)  fori  =  1, 2. 

Condition  (2)  implies  that  the  albedo  changes  arc  negligible  in  this  region,  so  that  the  first  term  may 
be  ignored. 


apn(V*£:{*i))  —  «P»»(p(*<)flp(p(*i))P**(*i))- 
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Figure  5.  One-Diniensio/iaJ  Example.  The  top  figure  illustrates  a  slice  of  the  surfaw.  contaii^g 
two  inflection  points.  The  second  figure  Uluslrates  the  first  derivative  of  the  surface  funrtton  The 
two  inflection  points  of  the  surface  correspond  to  extrema  in  the  first  derivauve.  TTic  third  figure 
illustrates  the  second  derivative  of  the  surface  function.  The  (wo  inflections  in  the  surface  corres^nd 
to  zero-crossings  in  the  second  derivative.  The  bottom  figure  illusUates  the  third  den vauve  of  the 
surface  function.  Between  the  points  corresponding  to  the  two  inflection  points  of  the  surface,  the 
third  derivative  contains  a  zero-crossing.  (From  Crimson  I1981b0. 
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Condition  (5)  implies  that0pn(p)  =  1.  Observing  that 

agn(xy)  =  8gn{x)  8gn[y), 

the  sign  of  the  convolved  intensity  function  at  the  siirfiKC  inflection  points  is  given  by 

8gn{V^E{xi))  =  8gn(Rp(p{Xi))p„{Xi)). 

Condition  ^3)  implies  that/ip  docs  not  change  sign  in  this  region  of  tlic  surface  and  hence 

8gn{Rp{p{xi)))  =  «ffn(R,,(p{x2))). 

Note  that  Px  =  0  at  x^,  x^.  The  fact  tliat  there  arc  exactly  two  inflections  on  the  surface  implies  that 
Px  ^  0  over  die  neighborhood  (ii ,  xx).  Ihus. 

affn(pxx(xi))  ^  apn(pxx(x2)). 


and  thus 

«gn(V^E(ii))  ^  8gn(V*E(a!2)) . 

ilic  second  differentiability  of  the  surface  implies  that  there  exists  a  point  x'  G  (xi.x^)  such  that 
V2/(a:')  =  0.| 

'nic  coiurapositive  of  this  theorem  states  something  important  about  the  possible  surfaces  which 
can  fit  the  known  depth  information.  Specifically,  given  the  conditions  of  the  theorem,  if  there  is  a 
set  of  known  depUi  points  to  which  a  surface  is  to  be  fiu  there  cannot  be  two  or  more  inflections  in 
die  surface  between  any  two  zero-crossing  points.  If  there  were,  there  would  have  to  be  an  additional 
/cro-crossing  there  as  well. 

Corollar}'  0. 1:  Suppose  one  is  given  a  set  of  known  depth  points  at  a  set  of  zero-crossings,  along  a 
developable  surface.  If  the  albedo  p  is  non-zero  and  the  reflectance  map  R  is  not  constant  for  this  region 
of  the  surface,  then  the  surface  cannot  contain  two  or  more  inflection  points  between  any  pair  of  adjacent 
zero-crossings.  | 

Tlicrc  arc  many  ways  of  extending  this  result  Some  arc  indicated  in  the  appendix.  The  most 
important  extension  is  to  consider  general  surfaces,  rather  than  developable  ones.  We  make  this 
extension  in  the  following  sections. 


I 
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4.3  The  Analytic  Argument 

Take  a  planar  slice  of  some  surface  /.  normal  to  the  image  plane,  along  stnnc  direction  (given  for 
example  by  the  angle  a  between  tlie  r  axis  and  the  direction  of  the  slice),  lliis  is  illustrated  in  l-'igurc 
6.  At  each  point  along  the  resulting  curve  C,  one  may  associate  a  surface  orientation,  or  giadicnt, 
given  by  the  pair  of  partial  derivatives,  p(i,  y)  =  df{x,  y)/dx,  and  q{x,  y)  =  df(x,  y)/dy.  I'hus,  one 
may  construct  a  two-dimensional  space  spanned  by  a  coordinate  system  with  axes  given  by  p  and  q, 
tlie  gradient  space  introduced  by  Hulfman  |I971]  and  used  by  Mackworih  [1973],  and  by  Horn  [1977J 
to  relate  the  geometry  of  image  projections  to  tlie  radiometry  of  image  formation.  Ihe  curve  obtained 
by  the  planar  intersection  of  the  surface  transforms  into  a  new,  parametric  cuive  in  gradient  space; 

N/(0  =  {P{<),9(0.-1} 

In  fact,  this  curve  corresponds  to  the  mapping  of  the  normal  to  the  surface,  as  one  moves  along  the 
planar  slice;  tlie  subscript  /  is  used  to  indicate  that  this  is  the  normal  to  the  surface  /. 

Because  the  albedo  is  roughly  constant,  the  image  irradianccs  are  determined  by  tlie  reflectance 
map  R{p,  q).  Thus,  the  irradiances  may  be  related  to  Horn’s  reflectance  map.  in  which  one  considers 
the  surface  defined  by  H{p,  q]  in  gradient  space.  Hence,  the  curve  C  on  the  original  surface  will  map 
onto  a  curve  N/  in  gradient  space  and  this  may  be  projected  onto  the  surface  /?(p,  q)  to  obtain  a  new 
curve  C'[t).  At  each  point  t  along  the  parametric  curve,  the  corresponding  irradiance  is  given  by 
/f(p((),  q(t)),  (sealed  by  the  constant  p,  which  we  will  ignore  without  loss  of  generality). 

We  are  interested  in  the  conditions  on  the  original  surface  and  the  reflectance  surface  that  will 
cause  a  zero-crossing  in  the  second  directional  derivative  of  the  image  irradiances.  (By  reflectance 
surface,  we  mean  the  reflecutnee  map  R(p,q)  considered  as  a  surface  in  gradient  space.)  Let  v  be  a 
vector  in  three-space  with  direction  cosines  cosa„  cosoy,  and  cosa^  (the  direction  cosines  refer  to 
the  cosine  of  the  angle  made  by  the  vector  and  the  coordinate  axes  of  the  space).  I’hus,  a  unit  vector  in 
the  direction  of  v  is  denoted  by 

V  =  co8axi-{-‘Co>Ovj  +  co>o>lt 

where  i,  j,  k  are  unit  vectors  along  the  coordinate  axes.  The  directional  derivative  of  $(r),  a  function  of 
a  vector  r,  is  the  rate  of  change  of  4>  with  distance  «  along  the  direction  v  and  is  denoted  by 

(V  .  V)*  =  COaO,—  +  -f  COBO,  —  . 


I 
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Figure  6  The  Generation  of  Image  Inadiance.  The  curve  C  is  generated  by  uilcing  the  intersection 
of  a  plane  normal  to  the  image  plane  with  the  surface.  The  direction  of  the  plane  is  given  by 
the  angle  a  it  makes  with  the  x-axis.  By  taking  the  surface  normal  at  each  point,  this  curve  C 
can  be  mapped  into  gradient  space,  resulting  in  a  parametric  curve  N/.  Furthermore,  this  curve 
can  be  projected  onto  the  reflecuuice  map  R  (which  defines  a  surface  on  p-q  space)  to  obtain  a 
parametric  curve  C'  along  which  the  irradiance  is  given  by  the  corresponding  value  of  R.  (From 
Crimson  (1981bD. 
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If  tlic  dircclion  vector  is  constrained  to  lie  in  Uic  i-y  plane,  then  cos  =  0  and  cos  Oy  =  sin  a^. 

Now  consider  a  planar  slice  no'mal  to  tlic  image  along  the  direction  v  and  constmet  die 
parametric  normal  curve  N/  in  p  —  9  space.  'Fhen  the  directional  derivative  of  irradiancc  along  this 
slice  is  given  by 

(y  •  V)E  =  fly((y  •  V)p)  +  rt,((y  •  V)g). 

litis  can  be  rewritten  as 

(v  V)E  =  VR  (y  V)N/ 

showing  that  the  directional  derivative  along  the  slice  is  given  by  the  projection  of  the  gradient  of  the 
reflectance  surface  R  onto  the  tangent  of  the  parametric  p-q  curve  N/.  Now,  if  there  arc  two  points 
t\,i2  such  titat  tills  dot  product  vanishes  at  those  points,  then  Rollc's  theorem,  and  tltc  assumption 
tJiat  the  surface  /  is  twice  continuously  dilTcrcntiablc  would  imply  the  existence  of  a  /cro-crossing  in 
the  second  directional  derivative  at  some  point  between  and  tj.  llius,  to  evaluate  the  probability 
of  a  zero-crossing  in  some  region  of  the  image,  we  need  to  consider  the  probability  of  the  vectors  VR 
and  (y  •  V)N/  being  orthogonal. 

Ihcrc  arc  two  eases  to  consider.  The  first  is  when  the  gradient  of  R  vanishes  at  some  point,  cor¬ 
responding  to  an  extremum  in  the  reflectance  surface.  If  we  assume  that  over  all  possible  viewpoints, 
VR  is  uniformly  distributed,  then  the  probability  of  VR  =  0  for  some  point  (p(f),  q(t))  on  the 
parametric  curve  N/(<)  is  given  by  the  arclcngth  of  the  curve 

/[(p.)"  +  («)*]^<. 

where  the  subscript  t  indicates  partial  difTcrentiation  with  respect  to  the  parameter  (. 

The  second  case  is  when  the  gradient  of  iZ  is  non-zero  and  the  two  vectors  arc  non-trivially 
orthogonal.  Consider  first  the  one-dimensional  case  in  which  two  inflections  in  a  developable  surface 
imply  the  existence  of  a  zero-crossing  (as  shown  in  Theorem  0).  Since  in  this  case  f  =  c,  the 
parametric  curve  N/{()  in  p-q  space  is  parallel  to  the  p-axis,  and  reverses  direction  at  each  of  the 
inflection  points.  At  these  points,  the  curvature  of  the  parametric  curve  is  essentially  infinite,  and  the 
tangent  can  be  considered  to  sweep  through  all  possible  directions.  As  a  consequence,  the  tangent 
must  be  orthogonal  to  VR  for  one  of  those  directions,  and  hence  (v  •  V)E  is  zero  at  each  point  of 
infinite  curvature.  Thus,  Rolle’s  theorem  guarantees  a  Kro-crossing  along  the  slice  of  the  ima^. 


THE  ANALYTIC  ARGUMENT 


20 


Now  suppose  that  q  ^  c  but  changes  slowly.  In  this  ease,  the  points  of  infinite  curvature 
baome  points  of  high  curvature.  Here,  the  tangent  to  tlic  parametric  curve  sweeps  through  a  large 
range  of  directions.  If  we  assume  tliat  the  second  order  partial  derivatives  of  die  reflectance  surface  R 
are  small,  tlien  over  a  small  region  of  p-q  space,  VR  is  nearly  constant.  Hence,  in  die  region  of  the 
points  of  high  curvature,  the  direction  of  V/l  is  nearly  constant,  while  the  direction  of  die  tangent  to 
N/  sweeps  through  a  large  range  of  directions.  Thus,  at  points  of  high  curvature,  die  probability  of 
(v  V)L  -  0  is  high.  If,  on  the  other  hand,  the  parametric  curve  is  locally  a  straight  line,  the  tangent 
is  constant  and  the  probability  of  the  directional  derivative  vanishing  is  small.  Hius,  along  a  slice  of 
die  image,  the  probability  diat  (v  ■  V)N/  and  Vft  arc  non-uivially  orthogonal,  under  the  as.suniption 
diat  R,,p,  Rf,^  and  R„  arc  small,  is  monotonically  related  to  the  integral  of  square  curvature 

f  —  qiPit)^ 

'  {Pi+q'i)^ 

where  again  (he  subscript  i  indicates  partial  differentiation  with  respect  to  die  parameter  t. 

Ihus,  w  e  can  combine  the  above  discussion  into  die  following  result 
Theorem  /  (Surface  Consistency  Theorem):  Consider  sotne  region  of  a  surface 

tfthe  reflectcnce  surface  R  is  isotropic,  its  normal  VR  is  uniformly  distributed  and  the  second  partials 
Rjip,  Rpq,Rqri  orc  Small,  then  the  probability  of  a  zero-crossing  in  (v  •  V)^E  for  some  direction  v,  with 
associated  directional  angle  a,  is  a  mono  tonic  fimetion  of 


and  of 


f  f 

J  hJt  (p2  +  q2)J 


where  the  integral  with  respect  to  t  is  taken  along  the  parametric  curve  obtained  by  slicing  the^  image  in 
some  direction  and  the  integral  with  respect  to  'ih  is  taken  overall  such  slices  of  the  image  region  ?)l).| 


This  theorem  has  the  immediate  corollary  of  specifying  the  "best"  surface  to  fit  through  a  set  of 
known  points. 
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Corollary  /./;  The  xurface  f  most  consisteni  with  a  set  of  known  points  provided  by  the  vistinl 
system  (e.g.  from  stereo)  is  that  which  is  least  likely  to  introduce  zero-crossings  not  present  in  the  primal 
sketch.  By  the  Surface  Consistency  Theorem,  this  surface  minimizes  arclengih  of  the  parametric  normal 
to  the  surface  along  each  direction  and  curvature  of  the  parametric  nonnal  along  each  direction,  subject 
to  passing  through  the  known  poinls-t 

Finally,  wc  would  like  to  obtain  a  simple  expression  for  lliis  surface.  Note  that  tlie  curvature  of  a 
curve  at  a  point  is  simple  Uic  rate  of  change  of  its  tangent  at  that  point,  niiis.  tlic  curve  with  minimum 
integral  square  curvature  is  one  willi  a  Lingeni  as  nearly  constant  as  possible,  relative  to  titc  constraints 
in?p<)scd  by  tlie  known  points.  Next,  to  minimi/.e  arclenth.  wc  require  the  curve  of  constant  tangent  to 
have  as  small  a  tangent  as  possible.  This  can  be  done  by  find  the  curve  which  minim/.ics 

which  is  equivalent  to 

J  (Pj-  cos  a  +  Pj/  sin  a)^  +  (Bx  cos  a-j-qy  sin  a)^  dl. 

Integrating  this  form  over  all  directions  a,  and  using  the  fact  that  Py  —  qj  (since  /  is  assumed  to  be 
twice  continuously  differentiable),  wc  obtain 

As  a  consequence,  wc  obtain  the  following  result 

Corollary  1.2:  The  surface  f  which  “best”  fits  a  set  of  known  data  in  a  region  91  of  the  image  is 
that  which  passes  through  the  known  points  and  minimizes 

f  f^fL  +  Vlv  + fly 

It  is  interesting  to  note  that  this  expression  for  “best"  surface  interpolation  as  the  surface  which 
minimizes  the  integral  of  quadratic  variation  has  been  derived  elsewhere  from  an  analysis  of  fitting 
this  clastic  plates  to  the  known  points  (Crimson,  1981b,  1982], 

Three  assumptions  were  made  in  the  statement  of  the  surface  consistency  theorem.  One  was 
that  tile  reflectance  surface  R  is  isotropic,  a  second,  that  its  normal  VR  is  uniformly  distributed. 
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and  a  tliird,  that  the  second  order  partial  derivatives  of  R  arc  small.  What  do  these  assumptions 
imply  about  the  imaging  situation?  Hie  assumption  of  an  isotropic  reflectance  function  is  minor. 
'I1ic  assumption  of  a  uniform  distribution  of  the  reflectance  normal  is  stronger.  Although  a  stronger 
version  of  the  surface  consistency  theorem  can  be  proved  by  weakening  this  assumption,  we  can 
also  justify  tlie  assumption  by  an  appeal  to  the  “general  position”  principle  of  viewer  geometry  and 
illuminant  geometry.  'Hie  notion  of  general  position  requires  that  all  properties  of  the  observed  image 
irradian  -  be  roughly  independent  of  the  specific  viewer  position  or  illuminant  position.  In  other 
words,  a  small  movement  of  the  viewer  or  the  illuminant  should  not  grossly  affect  the  properties  of 
the  image  irradiances.  In  terms  of  the  reflectance  surface,  a  slight  alteration  of  the  viewer  position 
or  illuminant  position  will  generally  result  in  a  translation  (and  possibly  a  scaling)  of  the  reflectance 
surface  R  relative  to  the  p-q  coordinate  system.  Within  the  proof  of  the  theorem,  the  assumption 
of  uniform  distribution  is  used  to  argue  that  at  any  point  in  the  image,  the  reflectance  normal  VA 
is  equally  likely  to  be  found  in  any  direction.  Ry  applying  the  principle  of  general  position,  we  can 
loosely  argue  that  the  illumina- .  is  equally  likely  to  be  in  any  of  a  range  of  positions  and  th.it  the 
wewer  is  equally  likely  to  be  in  any  of  a  range  of  positions.  As  a  consequence,  tltc  rcncctance  normal 
corresponding  to  a  point  in  the  image  is  equally  likely  to  be  found  in  any  direction,  and  hence  is 
uniConnly  distributed.  'Ilie  assumption  of  small andR^^  indicates  that  the  theorem  may  not 
be  valid  for  highly  specular  surfaces,  but  the  assumption  is  reasonable  for  surfaces  that  arc  generally 
matte  in  nature. 

4.4  The  Importance  of  the  Surface  Consistency  Theorem 

We  began  this  paper  by  discussing  the  problem  of  creating  complete  surface  representations 
from  sparse  data  such  as  that  provided  by  stereo,  and  argued  that  additional  constraints  were  needed 
to  effect  such  a  surface  reconstruction.  The  above  corollary  suggests  that  one  such  constraint  has 
been  identified.  In  particular,  given  a  class  of  surfaces  that  pass  through  a  set  of  known  points,  the 
surface  which  is  most  consistent  with  the  imaging  information  is  that  which  is  least  likely  to  introduce 
additional  zero-crossings.  By  the  surface  consistency  thcorm.  we  see  that  this  surface  is  defined  as 
the  surface  tliat  minimizes  the  functional  of  quadratic  variation,  or  at  least  minimizes  some  measure 
which  is  monotonically  related  to  this  ffinctional.  In  {Crimson,  1982},  we  combine  this  fact  with 
additional  constraints,  in  order  to  develop  a  computational  theory  of  visual  surface  interpolation. 
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5.  Examples 

To  illusiratc  these  results,  we  show  how  in  (lections  in  the  surface  can  cause  /cro-crossings  in  the 
convolved  image,  for  dilTerent  light  source  posilions.  l-'igures  7,  8.  9  ainl  10  show  examples  of  a  one- 
dimensional  slice  of  a  developable  surface,  with  a  1  ambcrlian  rcncclanco  function  (a  perfectly  Jilfusc 
surface  which  rellccts  (lux  uniformly  in  all  diicctions),  and  the  irradiance  values  obtained  fordilfcrcnt 
positions  of  a  point  light  source.  Figure  7  indicates  die  sample  surfaces  and  the  rough  positions  of 
die  light  sources  for  llie  dilfcrcnt  examples.  Figures  8,  9  and  10  indicate  sample  surfaces  and  tlic 
corresponding  iriadiancc  profiles  for  different  positions  of  the  light  source,  as  indicated  in  Figure  7. 
I'hc  positions  in  the  irradiance  function  which  would  give  rise  to  a  /cro-crossing  in  die  convolved 
image  arc  indicated. 


6.  Summary 

in  this  paper,  we  have  seen  (he  need  for  interpolating  the  rosulLs  of  tJtc  .stereo  algorithm  to 
create  complete  surface  descriptions.  Dctcnnining  whicit  surface  to  fit  to  the  known  stereo  data  is 
at  first  sight  difftcult,  since  any  one  of  a  widely  varying  family  of  surfaces  could  be  used.  Wc  saw, 
however,  tiial  most  of  these  surfaces  can  be  discarded  as  possible  interpolations,  since  tlicy  contain 
rapid  fluctuations  in  surface  orientation  and  these  fluctuations  should  give  rise  to  additional  zero- 
crossings,  not  contained  in  tlic  stereo  data.  Such  zero-crossings  would,  »,if  course,  be  inconsistent  with 
the  imaging  data,  and  hence  such  a  surface  is  considered  to  be  unacceptable  as  a  possible  interpolation 
of  the  known  data.  Ihus,  wc  proved  that  the  surface  consisiency  constraint  restricts  the  problem; 

The  absence  of  /.cro-crossings  constrains  the  possible  surface  shapes 
To  make  tliis  constraint  precise,  we  outlined  the  physics  of  image  formation  and  outlined  a 
derivation  of  Horn’s  image  irradiance  equation.  Wc  then  showed  that  this  expression  accounts  for 
the  zero-crossings  of  the  Marr-Hildrcth  edge  detection  algorithm  in  the  case  of  surface  boundaries, 
surface  markings,  texture  changes,  and  so  forth.  'Fhc  equations  further  suggest  that  even  in  regions 
where  the  physical  factors  causing  the  previous  types  of  zero-crossings  are  constant,  there  may  still  be 
zero-crossings,  due  to  a  change  in  surface  shape.  Wc  used  ibis  fact  to  prove  the  surface  consisiency 
theorem  which  relates  tlic  probability  of  a  zero-crossing  in  the  second  derivative  of  an  image  to  the 


Figure  8.  Diamples  of  a  Surface  with  Two  Inflections.  The  lop  figure  shows  a  slice  of  a  surface. 
The  bottom  three  figures  indicate  irradiance  profiles  for  diflerent  positions  of  the  light  source. 
Note  that  in  all  cases,  there  are  sis  irradiance  inflections,  in  case  3,  the  irradiances  also  undergo  a 
self-shadowing,  where  the  irradiance  value  is  zero.  The  positions  in  the  imidiance  function  which 
would  give  rise  to  a  zeio-crossing  in  the  convolved  image  are  indicated.  (From  Crimson  (1981bD. 


I'tgure  V.  bxamples  ot  a  Surface  with  Une  Inflection.  The  top  figure  shows  a  slice  of  a  surface. 
The  bottom  three  figures  indicate  irradiance  profiles  for  diflerent  positions  of  the  light  source. 
The  positions  in  the  irradiance  fiinction  which  would  give  rise  to  a  zero-crossing  in  the  conHalved 
image  are  indicated.  (From  Crimson  [198ibD. 


Figure  10.  Examples  of  a  Surface  wilh  One  Inflection.  The  lop  flguTe  shoe's  a  slice  of  a  surface. 
The  hoiiom  three  figures  indicate  irradiance  profiles  for  diflereni  positions  of  the  light  source.  Note 
Hint  in  case  8.  the  irradiance  profile  undergoes  a  self-shadowing.  The  positions  in  the  irradiance 
function  which  would  give  rise  to  a  zero-crosiung  in  the  convolved  image  are  indicaied.  (From 
Crimson  [1981b]). 
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variation  in  the  original  surface.  The  surface  consistency  constraint  can  be  used  to  formulate  a  com* 
putational  theory  of  surface  interpolation  and  a  specific  algorithm  for  computing  the  interpolated 
surface  [Grimson,  1982]. 


7.  Appendix 

11  I  cm  1  proved  a  relationship  between  any  general  two-dimensional  surface  and  tiiC  prob¬ 
ability  of  a  ycro  crossing  occuring  in  tlic  corresponding  convolved  intensities.  Ilccausc  the  theorem 
dealt  with  any  surface,  the  relationship  was  somewhat  weak.  In  this  appendix,  a  set  of  alternate 
theorems  arc  given  which  apply  in  cases  where  the  surface  can  be  considered  roughly  cylindrical  (or 
developable).  Ilic  proofs  of  these  theorems  arc  included  for  completeness. 

Theorem  2;  Consider  a  portion  of  a  second  differentiable,  developable  surface  oriented  along  the  y 
axissucb  thatfjix,y)  =  g-=  c,  for  some  constant  c.  If  the  following  conditions  are  true: 

1.  The  surface  portion  contains  exactly  two  inflection  points  in  the  x  direction,  at  *|  anda^, 

2.  At  the  points  xi  and  *2.  normalized  changes  in  albedo  are  dominated  by  normalized  changes  in 
reflectance. 
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3.  The  reflectance  map  R  does  not  pass  through  an  extremum  in  this  region  of  the  surface, 

4.  The  reflectance  map  R  is  not  constant  over  this  region  of  the  surface, 

5.  The  albedo  p  is  non- zero, 

then  there  ex’sts  a  point  x\  <  a:'  <  *2  such  thatV^E{aT)  =  0. 

Proof.  Ilic  signuni  function  is  defined  by: 

f  1  if  »  >  0 
agn(x)  =  |  — 1  if  x  <  0 
(0  if  X  =  0 

From  the  derivatives  of  the  image  equation. 


=  (VV)R  +  2pfflpp,  +  p{Rpppl  -1-  Ryp„). 
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Ai  ihc  inflection  points  *i  and  12.  Pj(i,)  =  0.  Hence,  evaluation  of  tlie  equation  yields 

V^E{zi)  =  -f-  p(xi)n^{p{xi))pr^{xi)  fort  =  1, 2 

Condition  (2)  implies  tltat  the  albedo  changes  are  negligible  in  tliis  region,  so  that  tlie  firet  term  may 
be  ignored, 

=  sgn{p{xi)R^{p{xi))p^j[xi)). 

Condition  (S)  implies  that  a9n(p)  =  1.  Observing  that 

8gn{xy)  =  8gn{x)  agn{y), 

the  sign  of  the  convolved  intensity  function  at  the  surface  inflection  points  is  given  by 

8gn{V^E{xi))  =  8gn{Rp{p{xi))pj^{Xi)). 

Condition  (3)  implies  that  Rp  does  not  change  sign  in  this  region  of  the  surface  and  hence 

agn(Rp(p(xt)))  =  agn(Rp(p(x2))). 

Note  that  Pj  =  0  at  ii,  12-  The  fact  that  there  are  exactly  two  inflections  on  the  surface  implies  that 
p,  ^  0  over  the  neighbourhood  (ii,  *2).  Thus, 

and  thus 

agn(V^E(x,))  ^  agn(V^E(x2)) . 

The  second  differentiability  of  the  surface  implies  that  there  exists  a  point  s'  €  [xi,  zj)  such  that 
V2E(*0  =  O.| 

Corollary  2. 1;  Suppose  one  is  given  a  set  of  known  depth  points  at  a  set  of  zero-crossings,  along  a 
developable  surface.  If  the  albedo  p  is  non- zero  and  the  rrflectance  map  R  is  not  constant  for  this  regim 
of the  surface,  then  the  surface  cannot  contain  two  or  more  irfflection  points  between  any  pair  of  adjacent 
zero-crossingSi§ 
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Theorem  3:  Consider  a  portion  of  a  second  differentiable  developable  surface  oriented  along  the  y 
axis  such  thatfy{z,y)  =  c,  for  some  constant  c.  If  the  following  conditions  are  true: 

1.  The  surface  contains  exactly  one  inflection  point  in  the  x  direction  at  x\,  and  the  reflectivity  R 
achieves  an  extremum  at  the  point  *2,  xi  ^  *2, 

2.  At  the  point  x\  the  normalized  changes  in  albedo  are  dominated  by  normalized  changes  in 
reflectance  of  the  form, 

P  ^  R  ' 

and  at  the  point  Xj  the  normalized  changes  in  albedo  are  dominated  by  normalized  changes  in  reflectance 
of  the  form, 

-  ^ppPt 

p  ^  R  ' 

3.  The  reflectance  map  R  is  not  constant, 

4.  The  albedo  p  is  non-zero, 

then  there  exists  a  point  *1  <  z'  <  X2  such  that  »  0. 

Proof.  As  in  the  proof  of  the  previous  theorem,  at  the  point  xi, 

sgn(y'^E{xi))  ~  *9«(^p(p(®i))P*i(*i))- 

At  the  point  Xi,Rp  =  0so  that 

V^Eixi)  =  V^p{x2fll(p(xi))  4-  p(x2)flpp(p(*2))P*(**)- 

Condition  (2)  then  implies  that  at  this  point,  the  normalized  albedo  changes  are  dominated  by  the 
normalized  reflectance  changes, 

«gn(V*E(»2))  =  ayn(p(i2))agn(/lpp(p(*2)))apn(p*(*2)). 

Condition  (4)  implies  that  Bgn(p)  «  1,  so  that 


$gr^V^E{xi))  =  «»n(Rpp(p(»2)))*9»^j4(*a)) 
=»  •gn(Hpp(p(*2))) 
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riicre  arc  two  subcases.  In  the  first  subcase,  PxA^i)  >  0.  Since  there  is  only  one  inflection  point 
in  the  surface,  tltis  implies  tliat  p{x\)  <  p(x2).  Ihcn  Rp,,{p{x2))  <  0  implies  that 

^;.(p(*i))  >  =  0. 

Conversely, .R,,,,(p{i2))  >  0  implies  that 

=  0- 

In  citlicrcase, 

agn(flpp(p(x2)))  7^  sffn(Rp(p(i,))p*x(*i)). 

In  the  second  subcase,  suppose  that  Pxx{xi)  <  0.  This  implies  that  p[x\)  >  p(x2).  Ihen 
Rpp{j^xi))  <  0  implies  that 

^j.(p(*i))  <  ^p(p(*2))  =  0. 

Conversely, /?pp(p{a>j))  >  0  implies  that 

^(P(*i))  >  fl!p(p(*2))  =  0- 

In  either  case, 

affn(flpp(p(i2)))  9^  a<;n(flp(p{i,))px,(ii))- 

Thus,  we  sec  that 

agniy'^Eixi))  ^  agn{y’^E[3^)) 

and  as  before  titc  second  differentiability  of  the  surface  implies  that  there  exists  a  point  s'  €  (xi,  x^) 
such  that  V*i5(x')  =  0.  § 

Corollary  3. 1:  Suppose  one  is  given  a  set  of  known  depth  points  at  a  set  of  zero-crossings,  along  a 
developable  surface.  If  the  albedo  p  is  non-zero  and  the  reflectance  map  R  is  not  constant  for  this  region 
of  the  surface,  then  if  the  reflectance  map  R  passes  through  an  extremum,  the  surface  cannot  contain  any 
inflection  points  between  any  pair  of  zero-crossings.  | 

Theorem  4:  Consider  a  second  differentiable  developable  surface  oriented  along  the  y  axis  such 
thatfy(x,y)  =  c,  for  some  constant  c.  If  the  following  conditions  are  true: 
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/.  At  the  poinixi,  the  surface  becomes  self-shadowing,  thatisli{xi)  =  0, 

2.  The  reflectivity  R  achieves  an  extnmum  at  the  point  xi.  xi  Xi. 

3.  At  the  point  xj  normalized  changes  in  albedo  are  dominated  by  normalized  changes  in  reflectance. 


w 

< 

RppPx 

p 

R 

4.  The  reflectance  map  R  is  not  constant  over  this  region, 

5.  Th^  albedo  p  is  non- zero. 

then  there  exists  a  point  Xi  <.xi  such  that  =  0. 

Proof,  llic  fact  that  the  surface  becomes  self-shadowing  implies  that  there  is  a  region  of  the 
surface,  beginning  at  xi,  such  that  R  is  constantly  zero.  The  fact  that  there  is  an  extremum  in  R 
fur  some  other  point  implies  that  tlie  intensity  fiinction  must  be  concave  down  in  the  region  of  the 
extremum  and  concave  up  in  the  region  of  self-shadowing.  There  must  be  an  inflection  point  in 
between  and  hence  there  must  be  a  point  x'  such  that  V*£7(i')  «  0.  g 

Theorem  5;  C nnsider  a  second  differentiable  developable  surface  oriented  r’ong  the  y  axis  such 
thatfy(x,y)  =  c,  for  some  constant  c.  If  the  following  conditions  are  true: 

/.  At  the  point  x\,  the  reflectivity R  achieves  an  it\flection point, 

2.  There  exist  points  xo<.xi  <.Xi  such  that  reflectance  changes  dominate  albedo  changes, 

3.  The  first  derivative  of  the  surface,  p.  is  monotonic  in  this  region,  (Le.  it  does  not  achieve  an 
extremum), 

4.  The  reflectance  map  R  is  not  constant  over  this  region, 

5.  The  albedo  p  is  nomzero, 

then  there  exists  a  point  *i  <  a/  <  xj  such  that  V^E{xl)  =  0. 

Proof.  The  proof  is  very  similar  to  the  previous  ones,  except  that  in  this  case,  by  condition  (2), 

»gn(V^E)  =  »gn{p)«gn{Rpp)$gn{pl) . 

at  the  points  xq,  xi-  Then  condition  (4)  implies  that 


apn(V*E)  =  tgn[Rpp). 


33 


Condition  (3)  implies  that 

p(®o)  <  p(*i)  <  p(*2) 

or 

p(i2)  <  p{x\)  <  p(aso) 
In  citltcr  cusc,  condition  (I)  tlicn  implies  that 


8gn{Rp,,{p(xo)])  ^  agT^Rpp(p{xi))). 

Hence, 

8gn(V^E(xi)))  7^  agn(V^E(x2)) , 

and  as  before,  the  second  differentiability  of  the  surface  implies  tliat  there  exists  a  point  G  [xo,  Xi) 
such  tliat  V'^E(jf)  =  0. 1 
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